OpenFabrics-suite-of-examples/ping-sr-4

The OpenFabrics suite of examples is code developed for the Programming
with OpenFabrics Software Training Course.

Copyright (c) 2011 OpenFabrics Alliance, Inc.  All rights reserved.

This software is available to you under a choice of one of two
licenses.  You may choose to be licensed under the terms of the GNU
General Public License (GPL) Version 2, available from the file
GNU_GPL_OFA.txt in the directory containing this source file, or the
OpenIB.org BSD license, available from the file BSD_for_OFA.txt in the
directory containing this source file.


This demo illustrates a ping-pong client server application using
send/receive to transfer data_size bytes.
It explicitly creates a cm event channel and a completion channel,
and puts both of them into O_NONBLOCK mode.
There are no threads.  Instead, the mainline uses unix poll()
on both channels to wait for something to happen.
It now empties the queue completely on each ibv_poll_cq, instead of
removing only 1 completion on each ibv_poll_cq.
We now call rdma_disconnect() when e receives an
(unexpected) RDMA_CM_EVENT_DISCONNECTED from the remote side, in order
to force a flush on any posted work requests.

The flow for the client program is:

		1. rdma_create_event_channel()
			creates a communication channel on which to report
			asynchronous cm events
		2. rdma_create_id()
			allocates a communication identifier rdma_cm_id
			add associates it with the cm event channel
			this is passed as a parameter to subsequent
			cm calls in order to identify the connection
		3. (rdma_)getaddrinfo()
			translates human-readable DNS name or IP address and
			port number into internal addressing form
		4. rdma_resolve_addr()
			does DNS lookup on server address and bind
			this rdma_cm_id to a local RDMA device
		5. await_cm_event()
		6. rdma_resolve_route()
			finds an RDMA route to the server
		7. await_cm_event()
		8. ibv_alloc_pd()
			allocates a protection domain (PD)
		9. ibv_create_comp_channel()
			creates a communicaton channel on which to report
			asynchronous completion events
		10.ibv_create_cq()
			creates a completion queue to hold pending events
		11.rdma_create_qp()
			creates a queue-pair (i.e., a send and a recv queue)
			and get it ready for sending and receiving
		12.ibv_req_notify_cq()
			requests notification when first completion is queued
		13.ibv_reg_mr()
			registers memory areas to be used for send and recv
		14.--create valid work requests and scatter-gather elements
			uses registered memory areas in send and recv
		15.rdma_connect()
			for RDMA_PS_TCP, initiates a connection request
				to remote server
			for RDMA_PS_UDP, initiates a lookup of the
				remote QP providing the datagram service
			allows user to specify operating parameters for
			this connection:
				max outstanding RDMA read/atomic ops to/from
					remote side
				use hardware flow control
				max retries on an error
		16.await_cm_event()
	loop
		17.ibv_post_recv()
			initiates a receive to catch next reply from the server
		18.ibv_post_send()
			initiates next send to the server
		19.await_completion_event()
			waits for send to complete
		20.await_completion_event()
			waits for receive to complete
	endloop
		21.rdma_disconnect() -- undoes step 15
			tells the cm to break the network connection
		22.await_cm_event()
			waits for the result from the cm
		23.ibv_dereg_mr() -- undoes step 13
			unregisters the memory areas used for send and recv
		24.rdma_destroy_qp() -- undoes step 11
			destroys queue pair
		25.ibv_destroy_cq() -- undoes step 10
			destroys the completion queue
		26.ibv_destroy_comp_channel() -- undoes step 9
			destroys the communication channel for completion events
		27.ibv_dealloc_pd() -- undoes step 8
			deallocates the protection domain
		28.rdma_destroy_id() -- undoes step 2
			destroys the cm_id
		29.rdma_destroy_event_channel() -- undoes step 1
			destroys the communication channel for cm events
